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EXISTENCE OF OPTIMAL CONTROLS FOR SINGULAR 
CONTROL PROBLEMS WITH STATE CONSTRAINTS^ 

By Amarjit Budhiraja and Kevin Ross 

University of North Carolina at Chapel Hill 

We establish the existence of an optimal control for a general 
class of singular control problems with state constraints. The proof 
uses weak convergence arguments and a time rescaling technique. 
The existence of optimal controls for Brownian control problems [14], 
associated with a broad family of stochastic networks, follows as a 
consequence. 

1. Introduction. This paper is concerned with a class of singular control 
problems with state constraints. The presence of state constraints, a key 
feature of the problem, refers to the requirement that the controlled diffusion 
process take values in a closed convex cone at all times [see (3)]. We consider 
an infinite horizon discounted cost of the form (4). The main objective of 
the paper is to establish the existence of an optimal control. 

Singular control is a well-studied but rather challenging class of stochas- 
tic control problems. We refer the reader to [7], especially the sections at 
the end of each chapter, for a thorough survey of the literature. Classical 
compactness arguments that are used for establishing the existence of opti- 
mal controls for problems with absolutely continuous control terms (cf. [8]) 
do not naturally extend to singular control problems. For one-dimensional 
models, one can typically establish existence constructively, by characteriz- 
ing an optimally controlled process as a reflected diffusion (cf. [2, 3, 15]). 
In higher dimensions, one approach is to study the regularity of solutions 
of variational inequalities associated with singular control problems and the 
smoothness of the corresponding free boundary. Such smoothness results 
are the starting points in the characterization of the optimally controlled 
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process as a constrained diffusion with reflection at the free boundary. Ex- 
cepting specific models (cf. [30, 31]), this approach encounters substantial 
difficulties, even for linear dynamics (cf. [32]); a key difficulty is that little is 
known about the regularity of the free boundary in higher dimensions. Al- 
ternative approaches for establishing the existence of optimal controls based 
on compactness arguments are developed in [12, 17, 25]. The first of these 
papers considers linear dynamics, while the last two consider models with 
nonlinear coefficients. In all three papers, the state space is all of W^, that 
is, there are no state constraints. It is important to note that in the cur- 
rent paper, although the drift and diffusion coefficients are constant, the 
state constraint requirement introduces a (nonstandard) nonlinearity into 
the dynamics. To the best of our knowledge, the current paper is the first to 
address the existence of an optimal control for a general multidimensional 
singular control problem with state constraints. While our method does not 
provide any characterization of the optimal control, it is quite general and 
should be applicable to other families of singular control problems (with or 
without state constraints). 

State constraints are a natural feature in many practical applications 
of singular control. A primary motivation for the problems considered in 
this paper arises from applications in controlled queueing systems. Under 
"heavy traffic conditions" , formal diffusion approximations of a broad family 
of queuing networks with scheduling control lead to the so-called Brownian 
control problems (BCP's) (cf. [14]). The BCP can in turn be transformed, 
by applying techniques introduced by Harrison and Van Mieghem [16] to a 
singular control problem with state constraints. We refer the reader to [1] for 
a concise description of the connections between Brownian control problems 
and the class of singular control problems studied in [1] and the current pa- 
per. In the Appendix, we indicate how the results of the current paper lead 
to the existence of optimal controls for BCP's. State constraints arise in nu- 
merous other applications: see Davis and Norman [10] and Duffie, Fleming, 
Soner and Zariphopoulou [11] (and references therein) for control problems 
with state constraints in mathematical finance. 

In Section 2, we define the singular control problem of interest. The 
main result of this paper (Theorem 2.3) establishes the existence of an 
optimal control. An important application of such a result lies in estab- 
lishing connections between singular control problems and certain optimal 
stopping/obstacle problems (see [29]). Such a connection was first observed 
by Bather and Chernoff [2] and has subsequently been studied by several 
authors [4, 18, 19, 30, 31] in one-dimensional and certain multidimensional 
models. The main result of the current paper is a key technical step (cf. [18]) 
in establishing equivalence with optimal stopping problems for a general 
class of singular control problems with state constraints. Such equivalence 
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results will be a subject of our future work. Connections between singu- 
lar control and optimal stopping, in addition to being of intrinsic math- 
ematical interest, have important practical implications. Singular control 
problems rarely admit closed form solutions and, therefore, numerical ap- 
proximation methods are necessary. Although numerical schemes for singu- 
lar control problems are notoriously hard, optimal stopping problems have 
many well-studied numerical schemes (cf. [22]). Exploiting connections be- 
tween singular control and optimal stopping is expected to lead to simpler 
and more efficient numerical solution methods. Indeed, in [6], we study a 
numerical scheme for a scheduling control problem for a two-dimensional 
queuing network by exploiting such connections. 

We now sketch the basic idea in the proof of Theorem 2.3. For a given 
initial condition w, we choose a sequence of controls {Un} such that the cor- 
responding cost sequence {J{w,Un)} converges to the value function V{w). 
The main step in the proof is to show that there is an admissible control U 
such that limmi J {w,Un) > J{w,U). For problems with absolutely continu- 
ous controls with a bounded control set, such a step follows from standard 
compactness arguments (cf. [8]); one argues that the sequence {Un} is tight 
in a suitable topology, picks a weak limit point U and establishes the above 
inequality for this U using straightforward weak convergence arguments. 
The key difficulty in singular control problems is proving compactness of 
the control sequence in a suitable topology; the usual Skorohod topology on 
D{[0,oo)) is unsuitable, as is suggested by the main result (Proposition 3.3) 
of Section 3. This result shows that for a typical discontinuous control U, 
one can construct a sequence of continuous controls {Un} such that the 
costs for Un converge to that for U ; clearly, however, Un cannot converge to 
U in the usual Skorohod topology on D([0,oo)). A powerful technique for 
bypassing this tightness issue, based on a suitable stretching of time scale, 
was introduced in [23]. Although such time transformation ideas go back to 
the work of Meyer and Zheng [27] (see also Kurtz [21]), the papers [23, 24] 
were the first to use such ideas in stochastic control problems. A similar 
technique was also used recently in [5]. This "time stretching" technique is 
at the heart of our proof. Time transformation for the nth control Ui is 
defined in such a way that, viewed in the new time scale, the process Un is 
Lipschitz continuous with Lipschitz constant 1. Tightness in D{[0, oo)) (with 
the usual Skorohod topology) of the time-transformed control sequence is 
then immediate. Finally, in order to obtain the candidate U for the above 
inequality, one must revert, in the limit, to the original time scale. This cru- 
cial step is achieved through Lemmas 4.2 and 4.3. The proof of the main 
result then follows via standard martingale characterization arguments and 
the optional sampling theorem. 

The proof of Theorem 2.3 is facilitated by the result (Proposition 3.3) 
that the infimum of the cost over all admissible controls is the same as that 
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over all admissible controls with continuous sample paths a.s. Although it 
may be possible to prove Theorem 2.3 without appealing to such a result, 
we believe that the result is of independent interest, and it simplifies the 
proof of the main result considerably. The main difficulty in the proof of 
Proposition 3.3 is that if one approximates an arbitrary RCLL admissible 
control by a standard continuous approximation (cf. [26]), state constraints 
may be violated. Ensuring that the continuous approximation is chosen in 
a manner that state constraints are satisfied is the key idea in the proof. 

The paper is organized as follows. We define the control problem and state 
the main result in Section 2. In Section 3, we characterize the value function 
as the infimum of the cost over all continuous controls. Section 4 is devoted 
to the proof of the main result. Finally, in the Appendix, we briefly describe 
connections with Brownian control problems and stochastic networks. 

We will use the following notation and terminology. The set of nonnegative 
real numbers is denoted by M+. For x G M'^, \x\ denotes the Euclidean norm. 
All vectors are column vectors and vector inequalities are to be interpreted 
componentwise. If ^ C M'^ and j4 is an n x d matrix, then AX = {Ax : x G 
X}. A set C C M*^ is a cone of M'^ if c G C implies that ac G C for all a > 
0. A function /:[0,oo) ^ is said to have increments in X if /(O) G X 
and f{t) — f{s) G X for all < s < t. A stochastic process is said to have 
increments in X if, with probability one, its sample paths have increments 
in X. Given a metric space E, a function / : [0, oo) ^ ii^ is said to be RCLL 
if it is right continuous on [0, oo) and has left limits on (0,oo). We define 
the class of all such functions by V{[0, oo) : E). The subset of ^^([O, oo) : E) 
consisting of all continuous functions will be denoted by C{[0,oo) : E). A 
process is RCLL if its sample paths lie in P([0, oo) :E) a.s. For T > and 
(p G I'([0, oo) : E), let = supo<j<7- l'^(^)l- We will denote generic constants 
in (0, oo) by c, ci, C2, . . .; their values may change from one theorem (lemma, 
proposition) to the next. 

2. Setting and main result. The basic setup is the same as in [1] . Let W 
(resp. U) be a closed convex cone of M'^ (resp. W^) with nonempty interior. 
We consider a control problem in which a p-dimensional control process 
U, whose increments take values in U, keeps a fc-dimensional state process 
W{t) = w + B{t) + GU{t) in W, where G is a fixed k xp matrix of rank k {k < 
p) and B is a A;-dimensional Brownian motion with drift b and covariance 
matrix S given on some filtered probability space {i},J-,{J-t},f). We will 
refer to $ = (O, T, {Tt}, P, B) as a system. We assume that n >V° / 0. 
Fix a unit vector vq G {GU)° n W. Select uq for which Guq = vq. We 
also require that there exist t)i G M'^, ■ui G W and cq G (0, oo) such that 



V ■ vi> aolv 



veGU 



w ■ vi> aolw 



(1) 



ueU. 
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The vectors uo,vo,ui and vi will be fixed for the rest of the paper. 

Definition 2.1 (Admissible control). An {jF^j-adapted p-dimensional 
RCLL process U is an admissible control for the system $ and initial data 
w (zW ii the following two conditions hold P-a.s.: 

(2) U has increments in U, 

(3) W{t) = w + B{t) + GU{t)eW, t>0. 

By convention, U (0—) = and VF(0— ) = w. The process W is referred to 
as the controlled process associated with U and the pair (W, U) is referred to 
as an admissible pair for <I> and w. Let A{w,^) denote the class of all such 
admissible controls. 

The cost associated with system initial data w; G W and admissible 
pair (W, U), U e A{w, <I>), is given by 



oo 



(4) J{w,U)=E e-^H{W{t))dt + E e~^^h-dU{t), 

Jo J[0,oo) 

where 7 € (0,oo), /i € and liW ^ [0, cxo) is a continuous function for 
which there exist constants Q,2; Q,3 ^ (0, 00) and G [0, 00), depending 
only on i, such that 

(5) Q,i|'u;|''^-Q,2<^(iL')<Q,3(kr' + l), weW. 

We remark that the assumption on i made above is weaker than that made 
in [1]. We also assume that h - U = {h ■ u:u ^ U} C M+. 

The value function of the control problem for initial data w & W is given 

by 

(6) V{w)=in£ mi Jiw,U), 

where the outside infimum is taken over all probability systems $. Lemma 
4.4 of [1] shows that V is finite everywhere. The following assumption will 
be needed for the main result of the paper: 

Assumption 2.2. (i) Either > or there exists ai G (0, oo) such that 
h- u>ai\u\ for all u gU. 

(ii) There exists cq G (0, oo) such that \Gu\ > cg\u\ for all uGlA. 

The following theorem, which guarantees the existence of an optimal con- 
trol for the above control problem, is the main result of this paper. The 
proof is postponed until Section 4. 

Theorem 2.3. Suppose that Assumption 2.2 holds. For all w G W, there 
exists a system ^* and a control U* G A{w, such that V{w) = J{w, U*). 
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3. Restriction to continuous controls. The main result of this section is 
Proposition 3.3, in which we show that in (6), it is enough to consider the 
infimum over the class of admissible controls with continuous paths. The 
use of continuous controls will play an important role in the time rescaling 
ideas used in the convergence proofs of Section 4. 

The proof of Proposition 3.3 involves choosing an arbitrary control and 
constructing continuous approximations to it. We show that the cost func- 
tions associated with the approximating controls approach the cost function 
of the chosen control as the approximation parameter approaches its limit. 
The main difficulty of the proof lies in constructing approximating controls 
so that state constraints are satisfied. Such a construction is achieved by 
means of the Skorohod map, which is made precise in the following lemma. 
We refer the reader to Lemma 4.1 of [1] for a proof. We recall that for T > 
and G r'([0,oo) :M''), denotes supo<t<T |'A(^)I• 
LEMMA 3.1. There exist maps P : P([0, oo) : M'^) ^ D([0, oo) : M'') and 
T : V^O, oo) : M'^) I?([0, oo) : M_|_) with the following properties. For any 4> € 
P([0,oo) :M'=) with (?i(0) e W, define r? = f ((^) and ip = P((/>). Then for all 
t>0: 

1. r]{t) G and i] is nondecreasing and RCLL; 

2. V'(i) G W and V'(i) = (pit) + voT]{t); 

3. // (Pit) E W for all t > 0, then r(</>) = (p and f (0) = 0. 

Furthermore, the maps P and P are Lipschitz continuous in the following 
sense. There exists k € (0, oo) such that for all (pi,(p2 G ^^([O, oo) rM*^) with 
(pii0),(p2i0) and allT>{), 

(7) \Ti<Pi) - Ti(P2)\*T + \ti<Pi) - f (<^2)It < ^\(Pi - Ht- 

Before stating the main result of this section, we present the following 
lemma which states that we can further restrict our attention to controls 
satisfying certain properties. The proof is contained in that of Lemma 4.7 
of [1] and is therefore omitted. 

Lemma 3.2. For w € W and a system <^>, let 

Miw, $) = ([/€ Aiw, ^): V r > 0, lim e"^*E|Ty(t)r = 

[_ >oo 
/•oo 

andE / e-^*|VF(t)rdt <oo, 

JO 

where W is the controlled process associated with U 
Then Viw) = inf$ ^niu£A'{w,^) Ji^^, U) . 
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Proposition 3.3. Let ^ be a system and w £W. Denote by A'^{w,^) 
the class of all controls U € A{w,^) such that for a.e. u, t^ Utioj) is a 
continuous map. Then 

(8) Viw) =inl inf J{w,U). 

Proof. Fix w eW and a system ^. Let U G A'{w,^) be such that 
J{w,U) <oo. Define U'^{t) = Y.o<s<t^U{s), where AU{s) = U{s) - U{s-) 
and [/^(t) = U{t) - W^it). That is, U'' is the continuous part and Lf^ is 
the pure jump part of the control U. Note that both processes are RCLL 
with increments in U. We construct a sequence of continuous processes to 
approximate as foUows. For each integer k>l and t € M+, set 

U^{t) = k t W^is) ds + k{l/k - t)+C/'^(0). 



'(^-l/fe)^ 

Note that for each k, is continuous with increments in U, and as k ^ oo, 

(9) Ui{t)^U'^{t) a.e. t G [0,oo), a.s. 

Also, from (1), it follows that for any function / with increments in ^, t>-^ 
f{t) ■ ui and 1 1— 5- • vi are nondecreasing functions. This observation 

implies that for all T > and < t < T, 

(10) aopmi < Um ■ ^1 < Vit) ■ u, < U\T) ■ ui < |[/'^(r)|, 

(11) ao\GUi{t)\<\GU'^{T)\, ao\GU'^{t)\ <\GU'^{T)\. 

Thus, by (9) and the dominated convergence theorem, for all p > 0, 
rT 

\Uf:{t)-U'^{t)\Pdt^O a.s. asA:^oo. 





This suggests that a natural choice for the approximating control sequence 
is {U'^ + U^}. However, this control may not be admissible, since the corre- 
sponding state process Wk, defined as Wk{t) =w + B{t) + GU^{t) + GU^{t),t > 
0, may violate state constraints. We now use the Skorohod map introduced 
in Lemma 3.1 to construct an admissible continuous control. Define, for 
t>0, mit)=t{Wk){t) and 

(12) Wk{t) = T{Wk){t) =w + B{t) + GU'{t) + GUm + GuoVk{t). 

Consider Uk = U^ + U^ + UQTjk- It is easily checked that Uk is continuous, 
{J^t}-adapted and has increments in U. Also, by Lemma 3.1, Wk{t) G W for 
all t > 0. Thus, Uk is an admissible control according to Definition 2.1. We 
will now turn our attention to the corresponding cost functions. We begin by 
proving that Wk{t) W{t) a.s. as /c — > oo. The main idea is to appeal to the 
Lipschitz property (7); however, (9) establishes only pointwise convergence 
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of Wk to W and so a direct application of (7) is not useful. Define, for each 
k>l, 

Wk{t) = k f W[s) ds + k(l/k - t)+VF(0). 

J{t~i/k)+ 

Since W{t) G W for all t > 0, it follows that Wk{t) G W for all t > and 
thus fik = TiWk) = 0. Recalling the definition of and using the Lipschitz 
property (7), we have, for T > and < i < T, 



\Wk{t)-Wk{t)\<K sup 

0<t<T I 



I B{t)-k /* B(s)ds 
I J(t-i/k)+ 



+ \G\ 



U%t)-k / 

(t-i/fc)+ 



From the sample path continuity of B and C/"^, the right-hand side of the 
inequality approaches almost surely as fc — s- oo. Next, since W has RCLL 
paths, Wk{t) W{t—) a.s. for every t > 0. Combining the above observa- 
tions, we have that as /c — > oo, 

(13) Wk{t) — > W{t) as /c — > oo for almost every t G [0, oo), a.s. 

We now show that the costs associated with controls Uk converge to the 
cost corresponding to control U. We first consider the component of the 
cost arising from i. Using (1), we have, along the lines of equations (4.10)- 
(4.12) of [1], that there exists ci G (0, oo) such that for < i < T < oo, 

(14) \GU''it)\ + \GU\t)\ + \W{t)\ < cii\w\ + \WiT)\ + \B\*t). 

Writing Wk = Wk — W + W and using Lemma 3.1, we have, for all > 1 
and < t < T < oo, 

(15) \Wk{t)\ < K{\GUi\*T + \GU%) + \W{t)\ < C2{\w\ + \W{T)\ + \B\*t), 

where the second inequality follows from combining (14) and (11). Recalling 
(5), we obtain, for some C3 G (0, 00), 

i{Wk{t)) < o,{\wr + \w{t)r + mir + 1)- 

Finally, since U G A'{w, $), we have, from the above estimate, (13) and the 
dominated convergence theorem, that as A; — > 00, 



00 roo 



(16) E/ e-^HiWkit))dt^E e~^H{W{t)) dt. 

Jo Jo 

We now consider the component of the cost function associated with h. 
Note that since E e~'^^h ■ dU (t) < J{w, U) < 00, we have that 

(17) e/ e-^%-dU{t)=-f f e-^^E{h-U{t))dt<oo. 

J[0,oo) J[0,oo) 
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Next, for t > 0, 

\Guomit)\ < C4i\w\ + \Wk{t)\ + \B{t)\ + \GU%t)\ + IGW'm 

<C5{\w\ + \Wk{t)\ + \W{t)\ + \B\;) 

<cei\w\ + \Wit)\ + \B\:), 

where the first inequahty fohows from (12) and (11), the second from (14) 
and the third from (15). Since % is nondecreasing, the above display imphes 
that \7]k\t < C7{\w\ + \W{t)\ + |S|i*). Thus, since U e A'{w,^), we have that 

(18) 7E / e-^\h-uo)r]kit)dt = E [ e'^^h ■ uq drjkit) < 00. 

J[0,oo) "'[0,00) 

Next, 

E / e-^*h-dUk{t) 

JlO,oo) 

= e( [ e~^%-dU''{t)+ I e-^%-dU^{t) 

\J[0,oo) J[0,oo) 

(19) +/ e-^"h-uodr]k{t)) 

J[0,oo) / 

= 7E f / e~^*/i • U^it) dt+ [ e-^'^h ■ U^{t )dt 

\J[0,oo) J[0,oo) 

+ / e"^% ■ uorjkit) dt) , 
J 10,00) / 

where the last line follows from using (18), noting that K{h- {Uj:{t) +U'^{t))) < 
E(/i • U(t)) and recalling that J{w,U) < 00. From (17), (18) and (19), it 
now follows that E /jq e~'^^h ■ dUk{t) is finite and equals 7E e~'^*/i • 
Uk{t)dt. 

Prom (9) and (13), we get that as /c — > 00, 

(20) (/i-iio)??fc(t) ^0 and h ■ U^{t) ^ h ■ U'^{t), a.e. t, a.s. 

Recalling that < C7(|tt;| + \W{t)\ + \B\l) and that U G ^'(u;,$), equa- 
tions (20) and (18) imply that as A; — > 00, 

(21) E / e"'^*/i-iiod%(t) ^0. 

J[0,oo) 

Since h-U^{t)<h- ^^{t) and E e"^*/! • C/°'(t) dt < J{w, U)<oo,we get, 
from (20), that as /c — > 00, 

(22) e[ e-^^h-dUi{t)^E [ e'^^h ■ U'^{t) dt. 

J[0,oo) J[0,oo) 
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Finally, taking limits as /c — > co in (19) yields 

(23) e[ e^'^^h-dUk{t)^E [ e~^^h-dU{t). 

J[0,oo) J[0,oo) 

Combining (16) and (23), we have J{w,Uk) J{w,U) as A: — > oo. This 
proves the result. □ 

4. Existence of an optimal control. In this section, we prove our main 
result (Theorem 2.3) which guarantees the existence of an optimal control for 
the control problem of Section 2. Fix w G W. From Proposition 3.3, we can 
find a sequence of systems with = (r2„, {.7>^(t)}, P.„, i?„) and a 

sequence of controls {Un} with Un S A^{w, ^n),n > 1, such that J{w, Un) < 
oo for each n and 

(24) V(w) = lim J(w,Un), 
where 

(25) J{w, Un) = E„ / e-^H{Wn{t)) dt + E„ / e-^'h ■ dUn{t) 

Jo J[0,oo) 

and Kn denotes expectation with respect to P„. Let Wn be the state process 
corresponding to Un, that is, 

(26) Wnit)=W + Bn{t) + GUn{t), 

with Wn{t) G W for ah t > 0. 

Time resettling . For each n > 1 and t>0, define 

(27) Tn{t)=t + Un{t)-Ui. 

Since ui ■ Un is continuous and nondecreasing, t„ is continuous and strictly 
increasing. Also, for < s < t, 

(28) Tnit)-Tnis)>t-S, Tn{t) - Tn{s) > ao\Un{t) - Un{s)\. 

The time-rescaled process is given by f„(t) = inf{s > 0:r„(s) > t}. Note 
that fn is continuous and strictly increasing. Also, t = f„(Tn(t)) = rn(f„(t)), 
Tnit) <t < Tn{t), and f„(s) < t if and only if T„(t) > s. 

We define the time-rescaled processes via Bn{t) = Bn{fn{t)),Un(t) = 
Unifnit)) and W'„(t) = Wnifnit)). From (26), for t > 0, 

Wn{t) = Wn{fn{t)) =W + S„(f„(i)) + GUn{fn{t)) 

(29) 

= W + Bn{t) + GUn{t). 

Also, from (28), for < s < t, 

(30) fn{t)-fnis)<t-S, ao\Un{t)-Un{s)\<t-S. 
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Let £ denote the space of continuous functions from [0, oo) to M}' x [0, oo) x 
M'^ X Z// X W, endowed with the usual topology of uniform convergence on 
compact sets. Note that for each n > 1, {Bn,fn, Bn,Un,Wn) is a random 
variable with values in the Polish space £. We next consider tightness of the 
family {(-B„, f„, f7„, W'n), n > 1}. 

Lemma 4.1. The family f„, [/„, W'n), n > 1} is tight. 

Proof. Clearly, {Bn} is tight. Tightness of {(f„, [/„)} follows from (30). 
Since Bn{t) is the composition of Bn{-) and t„(-), tightness of {Bn} follows 
from tightness of {(-B„,fn)}. Finally, tightness of follows from (29) 

and tightness of {(fn, C/n, -B„)}. □ 

Choose a convergent subsequence of {{Bn,fn,Bn,Un,Wn),n> 1} (also 
indexed by n) with limit {B' , f, 13, U, W) defined on some probability space. 
Clearly, B' is a (6, S)-Brownian motion with respect to its own filtration. 
By the Skorohod representation theorem, there exists a probability space 
(Q* ,¥*) on which are defined a sequence of processes {{B'^^fn, B'^, U^jW^), 

n> 1} and a process {B" ,f' , B' ,U' ,W'), such that {B'^,fl^, B'^Ml^Wl,) = 
{Bn,fn,Bn,Un,Wn), {B'\ f' , & ,U' ,W') = {B' , f , B ,U ,W) and (5;,f;,^;, 
Un,Wl^) {B" ,f' , B' ,U' ,W') almost surely as n^oo. To simplify nota- 
tion, we will assume (without loss of generality) that 

(31) {Bn,fn,Bn,Un,Wn) ^ {B',f,B,U,W) P*-almost surely as n^oo. 

The following lemma plays a central role in the time rescaling ideas used 
in this section: 

Lemma 4.2. Suppose that Assumption 2.2 holds. Then there exists a* G 
(0, oo) such that for all t>0, 

(32) limsupE„|CZ„(t)|"* < oo. 

Proof. Recall that liuin^oo J {w , Un) = V{w) < oo. From Assumption 
2.2, we have that either > or there exists ai G (0, oo) such that h ■ u> 
ai\u\ for all uGU. Suppose first that the latter condition holds. Then for 
aU t > 0, 

roo 

J{w, Un) > 7E„ / e-""h ■ Unit) dt > 7e-^(*+^)E„(/i • C/„(t)) 
Jo 

>7aie-T(*+i)E,|C/„(t)|. 
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Thus, in this case, (32) holds with a* = 1. Next, suppose that ai > 0. Prom 
Assumption 2.2 and (26), we have 

CG\Unit)\ < \GUn{t)\ < \Wnit)\ + |5„(t)| + \w\, 

which imphes that for some ci € (0, oo), 

\Un{t)r < Cl{\Wn{t)r + \Bn{t)r + iwD- 

Therefore, using moment properties of we have, for some C2 € (0, oo), 

(33) ^n\Un{t)r< C2{En\Wn{t)r + + !)• 

Combining the above estimate with (5), we get 

roo 

(34) hmsup / e-^'En\Un{s)\"' ds < oo. 

n— »oo JO 

Finally, 

Jo Jo 

Inequality (32) now follows, with a* = ae, by combining the above inequality 
with (34). □ 

The following lemma, a consequence of Lemma 4.2, gives a critical prop- 
erty of f : 

Lemma 4.3. Suppose that Assumption 2.2 holds. Then 

(35) f(t) ^ oo as t —> oo, P*-a.s. 

Proof. Pix M > and consider t G (M, oo). Since f„,(t) < M if and only 
if r„(M) > t, we have, by (27), 

{fn{t) < M} = {M + UniM) • ni > t} C {\UniM)\ >{t- M)}. 

Recall the constant a* in Lemma 4.2. The above relation and an application 
of Markov's inequality yield, for all t > M, 

P*[f„(t) < M] <P*[|C/„(M)|°* > (t - M)"*] < ^-_l-^E*|[/„(M)r*. 
Thus, by the weak convergence fn =^ f. 



lim f{t) < M 

t— >oo 



< lim limsupP*[f„(t) < M] 



t~^00 Yl ,.QQ 



< hm ^ , limsupE*|^4M)|"*. 

t^oo [t — M)°' n^oo 
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The right-hand side of the last inequahty is 0, by Lemma 4.2. Since M > 
is arbitrary, the result follows. □ 

We now introduce an inverse time transformation which allows us to revert 
back to the original time scale. For t > 0, define r(t) = inf{s > : t{s) > t}. 
The following properties are easily checked: 

• r(i) < oo a.s. for all i > (this follows from Lemma 4.3); 

• r is strictly increasing and right continuous; 

• T{i) > i > T'{i) and in particular, T{t) — > oo a.s. as t ^ oo; 

• < f (s) <t^O<s< r(t), and r(r(t)) = t, r(f(t)) > t. 

The time-transformed processes are defined as B*{t) = i3(r(t)), U*{t) = 
U{T{t)), W*it) = W{T{t)), t>0. By (31) and (29), we have W{t) = w + 
B{t) + GU{t) for all t > 0, a.s., which implies that 

W*{t) = W{T{t)) = w + B{T{t)) + GU{T{t)) =w + B*{t) + GU*{t). 

Note that U* is RCLL with increments in U and W*{t) G W for all t > 0. 

We next introduce a suitable filtration on {^l* ,J^* ,¥*). For i > 0, define the 
c7-fields P{t) = a{{B{s),U{s),W{s),f{s)),0 < s < t} and :Ft = T{t) = 
J-'{t+) y J\f , where M denotes the family of P*-null sets. Then {J-t} is a 
right-continuous, complete filtration. For any s,t > 0, {t(s) <t} = {f{t) > 
s} G ^{t). Therefore, since {Tt} is right continuous, r(s) is an {J^t}-stopping 
time for any s > 0. For each t > 0, define the <T-field = T*{t) = T{T{t)). 
Since r is nondecreasing, {!Ft} is a filtration. Clearly, B and U are {jFt}- 
adapted; therefore, B* and U* are {.Fj*}-adapted (cf. Proposition 1.2.18 of 
[20]). We show in Lemma 4.6 below that B* is an {.Fj* }-Brownian motion 
with drift h and covariance matrix S. Before stating this result, we present 
the following change of variables formula which we will use in the conver- 
gence analysis. We refer the reader to Theorem IV. 4. 5 of [28] for a proof. 

Lemma 4.4. Let a he anM^-valued, right- continuous function on [0,oo) 
such that a(0) = 0. Let c be its right inverse, that is, c{t) = inf{s > : a{s) > 
t}, t>0. Assume that c{t) < oo for all t>0. Let f be a nonnegative Borel 
measurable function on [0,oo) and let F he an M.^-valued, right- continuous, 
nondecreasing function on [0, oo). Then 



with the convention that the contribution to the integrals above at is 
/(O)F(O). In particular, taking F{s) = s,s>0, we have 



(36) 




(37) 




14 A. BUDHIRAJA AND K. ROSS 

Remark 4.5. Recall that Bn{t) = Bn{fn{t)). It follows from continu- 
ity and almost sure convergence of {Bn,fn,Bn) {B',f,B) that B{t) = 
B'{f{t)) a.s. Thus, B*{t) = B{T{t)) = B'{f{T{t))) = B'{t) a.s. In particular, 
B* is a (6, S)-Brownian motion with respect to its own filtration. The fol- 
lowing lemma shows that, in fact, B* is a Brownian motion with respect to 
the larger filtration {T^}: 

Lemma 4.6. B* is an {J^^} -Brownian motion with drift b and covari- 
ance matrix S. 

Proof. For any infinitely differentiable function /iM'^ — > M with com- 
pact support, define 

(38) Am=p,i.f(.) + ix:i:-«isS^-^<^)- 

1=1 * j=ij=i ' J 

where the entries of b are denoted bi and those of S are denoted aij. First, 
suppose that 

g{B{Sm),U{Sm),W{Sm),f{s 

m) ) Sm ^t,m — 







X f{B{t + s))- fiB{t)) - / Af{B{u)) df{u) 



for all s,t > 0, continuous bounded functions g (on a suitable domain), pos- 
itive integers g' > 1 and sequences {sm}- Define, for t > 0, 

Yf{t) = f{B{t)) - f Af{B{u))df{u). 
Jo 

Then by equation (39), Yy is an {^^'}-martingale and, therefore, Yj is also an 
{.Fj}-martingale. Recall that t{s) is an {jT^j-stopping time such that r(s) < 
oo a.s. for all s > 0. Since / and Af are bounded (by some c> 0), 

E*\Yf{Tm<E*\f{B{Tm\+E* r'^'\Af{B{u))\dT{u) 



<c + cE*\f{T{t))\=c{l + t). 
In addition, we have, for any T G (0, oo), 

E*[|i}(r)|i|,(,)>r}]<iE*[|^/(r)|%(T)<t}] 

<E 



\f{B{T))\+ \Af{B{u))\df{u)\l{,^^r)<t} 
<cE*[(l + f(r))l{^(r)<i}] 

<c(i + t)p*[f(r) <t]. 
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The last term above approaches as T ^ oo, by Lemma 4.3. Therefore, 
by the optional sampling theorem (cf. Theorem 2.2.13 in [13]), we have, for 
s<t, 

r[l>(r(i))|^*(s)] =r[l>(r(t))|.F(r(s))] =17(r(s)), 
that is, Yj(r(t)) is an {.F^*}-martingale. Now, 

roc 

Yfirit)) = fiBirm - / ^/(B(n))l|o<u<r(t)} 



= f{B*{t)) - Af{B{T{u)))l^o<riu)<rit)} du 

= f{B*{t))- f Af{B*{u))du, 
Jo 

where we have used Lemma 4.4 and the fact that r is strictly increasing. 
Thus, 

rt+s 



E* 



f{B*{t + s)) - f{B*{t)) - ' Af{B*{u)) du\T*{t) 



which is for any s,t>0, since Yf{T{t)) is an {^^*}-martingale. Therefore, 
B* is an {.F(*}-Brownian motion with drift b and covariance S. Hence, in 
order to prove the lemma, it suffices to prove (39). 

Recall that Bn is an {^„(t)}-Brownian motion with drift b and covariance 
E. Let / be as above and define Yf^n{t) = f{Bn{t)) - /g Af{Bn{u))du. Then 
Yf^n is an {.Fn(t)}-martingale for each n > 1. 

Fix t > and note that {fn(s) <t} = {r„(t) > s} = {t + Un{t) • ni > s} G 
Tn{t) for all s > 0, n > 1. Thus, for each s > 0, fn{s) is an {J>i(t)}-stopping 
time. Define Yf,n{t) = Yy „(f„(t)) for t > 0. Since fn{t) is an {jF„(t)}-stopping 
time bounded by t, we have, by the optional sampling theorem (see Problem 
1.3.24 in [20]) that for any s > 0, 

IEn[^/,n(t + S)|^n(r„(t))] = E„[y/,„(f„(t + s)) (f„ (t) )] 

= i7,n(f„,(t)) = y/,„(t). 

This implies that for any bounded .7>i(f„(t))-measurable function 
(40) EnlUYfAt + YfA^)}] = 0- 

Now, for any s <t, the random variables i?„(f„(s)), ?7„,(-f„(s)) and Wn{Tn{s)) 
are ^,i(f„(s))-measurable (cf. Proposition 1.2.18 in [20]). Also, fn{s) is 
•^n('?'n('S))-measurable (cf. Problem 1.2.13 in [20]). Since fn{s) < fn{t) for 
s < t, we have jr„(f„(s)) C Tn{Tn{t))- Thus, 

g{Bn{fn{Sm)), Un{fn{Sm)), Wn{fn{Sm)), fn{Sm), < < t, m = l,...,q), 
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is JP"„(f„(t))-measurable for all bounded continuous functions g (defined on 
an appropriate domain), positive integers q>l and sequences {sm}- There- 
fore, using (40) and recalling our use of the Skorohod representation theorem 
above (31), we have 

^*[g{Bn{Srn):Un{Srn),Wn{Srn),Tn{Sm),^< Sm < t, m = 1, . . . , q) 

(41) 

x{Yf^n{t + s)-Yf^^{t)}]=0. 
Another application of Lemma 4.4 shows that 



Af{Bn{u))dfn{u) = 
This implies that 

Yf,n{t)=Yf,n{rn{t)) 

(42) =/(i?„(f„(t))) 



r„{t) 



Af{Bniu))du. 



Af{Bn{u))du 

f{Bn{t))- I Af{Bn{u))dfn{u). 

Combining (41) and (42), we have 



(43) 



g{Bn{Sm),Un{Sm),Wn{Sm),Tn{Sm),^< Sm < t, 171 = I, . . . , q) 



x\f{Bnit + s))-f{Bnit)) 



t+s 



Af{Bn{u))dfn{u) 



0. 



Finally, recall that (5„, f„, [/„,#„) ^ (5', f, i?, [/, #) P*-a.s. as n ^ 
oo. Thus, in particular, as n ^ oo, J^'^^ Af{Bn{u)) dfn{u) converges almost 
surely to Jl~^^ Af{B{u)) df{u) (cf. Lemma 2.4 of [9]). An application of the 
bounded convergence theorem now yields (39) on taking n — > oo in (43). 
This completes the proof. □ 



As an immediate consequence we have the following: 



Corollary 4.7. Let ^* = {n*,T*,F*,{T^},B*). Then U* e A{w,<^*). 

We now show that U* is an optimal control by studying convergence of 
the cost functions J{w, Un), thus completing the proof of the main result. 

Proof of Theorem 2.3. Let {Un} and U* be as above. By Lemma 4.4, 
we have that the cost corresponding to the admissible pair {Wn, Un) is given 
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by 

POO roo 

J{w, Un) = En / e-^H{Wn{t)) dt + jEn / e'^'h ■ Unit) dt 
Jo Jo 

Jo 

(44) + 7E„ / e-^^"(*)/i • Unifnit)) df„(t) 

Jo 

roo 

= E* e-"'^"^'h{Wn{t))dfn{t) 

Jo 

roo 

+ 7E* / e-^^"^'h-Un{t)dfn{t). 



Since (f„, #„) {f,U,W) P*-a.s., we have (cf. Lemma 2.4 of [9]), for 
all u > and N>1, 

[0,u) 

[ [N Ae-^^^^h{W{t))]df{t), P*-a.s. 

J[0,m) 

as n — > 00. Thus, we have, P*- almost surely, 

rca 

liminf / e-^^"W£(t^„(t))df„(t) 



n— >oo 



> / [N Ae~^^^^k{Wit))]df{t). 
Jo 

Taking limits as ^ 00 and u ^ 00 in the above inequality yields 
liminf/ e-^^"^^h(WJt)) dfnit) 

roo 

> / e-^'^^'h{Wit))df{t), 



(45) 
Similarly, 



-a.s. 



liminf 7 / e-^^"(*)/i • Ujt) dfjt) 

J[0,oo) 



>7 / e-T^W/i- ?7(t)df(t), 

J[0,oo) 

Combining (24), (44), (45) and (46), we have 
V{w) = liminf J{w, Un) 



-a.s. 
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lim inf 

n— >oo 



+ 7E*/ e-^^"^'^h-Un{t)dfn{t) 
Jo 

>E*liminf / e-^^"'^*k(WJt)) dfJt) 

+ 7E* lim inf / e-^"""(*)/i • UJt) dfJt) 

>E* e-^'^^'h{Wit))dT{t) 
Jo 

+ 7E*/ e-^^^^'>h-U{t)df{t). 



Applying Lemma 4.4 to the last line above and recalling that W*{t) 
W{T{t)) and U*it) = UiT{t)) yields 



00 ;>oo 



V{w) > E* / e-^H{W*{t))dt + 7E* / e~^^h ■ U*{t) dt 



The quantity on the right-hand side above defines the cost function J{w, U* 
for the admissible (by Corollary 4.7) pair {W*,U*). Thus, we have V{w) - 
J{w,U*) and, hence, U* is an optimal control. □ 



APPENDIX 

In this section, as an application of Theorem 2.3, we prove the existence of 
an optimal control for a family of Brownian control problems. Such control 
problems (cf. [14]) arise from formal diffusion approximations of multiclass 
queuing networks with scheduling control. Here, we do not describe the 
underlying queuing problem, but merely refer the reader to [5], where details 
on connections between a broad family of queuing network control problems 
and Brownian control problems can be found. Our presentation of BCP's is 
adapted from [16]. 

Let $ = {^},J-,{J-t},^,B) be a system, where B is an m-dimensional 
Brownian motion with drift b and nondegenerate covariance matrix S. The 
problem data of the BCP consists of an m x n matrix R, a p x n matrix K 
(referred to, respectively, as the input-output matrix and the capacity con- 
sumption matrix) and an initial condition q € M™. The matrix K is assumed 
to have rank p {p<n). 



Definition A.l (Admissible control for the BCP). An {jT^j-adapted 
n-dimensional RCLL process Y is an admissible control for the BCP as- 
sociated with the system <I> and initial data q € M™ if the following two 
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conditions hold P-a.s.: 

U{t) = KY{t) is nondecreasing with U{0) > 0, 
Q{t) = q + B{t)+RY{t)>0, t>0. 

Denote by A{q, <I>) the class of all admissible controls for the BCP asso- 
ciated with $ and q. The goal of the BCP is to minimize the cost function 

J{q,Y)=E e-"'*i{Q{t))dt + E e-"'* h ■ dU (t) , 

Jo J[0,oo) 

where 7 € (0, 00), h G and I : — > [0, 00) is continuous. The value func- 
tion for the BCP is V{q) = inf |, inf yg_^(.^ J{q,Y). 

Under a continuous selection condition (see [16] or equation (3.3) of [1]), 
the BCP introduced above can be reduced to an equivalent control prob- 
lem of the singular type (with state constraints). This reduction, referred 
to as the "Equivalent Workload Formulation" (EWF), is the main result of 
[16]. Subject to further conditions, this singular control problem with state 
constraints is of the form studied in the current paper. Such sufficient con- 
ditions are presented in Section 3 of [1]; however, we list them here for the 
reader's convenience. Let i be linear and nonnegative on and assume that 
it vanishes only at zero. Define B = {x gR"- : Kx = 0}. Let TZ = RB cM'''' 
and denote the dimension of TZ by r. The dimension of = TZ^ is then 
k = m — r. Let M be any k xm matrix whose rows span M. By Proposition 
2 of [16], there exists a kxp matrix G which satisfies MR = GK. In general, 
the choice of G is not unique. We assume that the matrices M and G are of 
full rank and have nonnegative entries. We further assume that each column 
of G has at least one strictly positive entry. These assumptions are satisfied 
for a broad family of controlled queuing networks (see Section 3 of [1] and [5] 
for examples). Under these assumptions. Theorem 2.3 leads to the following 
result: 

Theorem A. 2. For every q G W^, there exists a system <I> and Y G 
A{q,^) such that J{q,Y) = V{q). 

Remarks on the proof. The proof is an immediate consequence of 
Theorem 2.3 and Proposition 3 of [16]. The latter proposition shows that 
for any admissible control for the EWF, there exists a control for the BCP 
(and vice versa) such that the costs coincide. Since an EWF under the above 
assumptions is a control problem of the form formulated in Section 2, the 
existence of an optimal control for the EWF follows from Theorem 2.3. Using 
the equivalence result in Proposition 3 of [16], one then obtains an optimal 
control for the BCP. 
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